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AESTRACT 



To assess behavioral change in the speech of 
students, a test was developed which used the prior subjective 
experiences cf teachers as a foundation ter comparing student 
behaviors, and objectivized these experiences by developing a rating 
system upon which teacher observations could be scaled. Twenty 
raters — 10 experienced and 10 inexperienced — were selected, trained, 
and divided into twe groups. Sets ot four raters then observed 
randomly-selected classrooms at the elementary and secondary levels 
for two rating periods. Results of data indicated that no statistical 
differences existed between judges* measurements in any of the class 
levels and that judges* measurements were reliable indices of 
communicative behavior. It was concluded that this method (1) can 
provide educators with a reliable means of assessing the attainment 
of behavioral change, (2) may functicn more reliably on the secondary 
level than on the elementary level, (3) prepares outside observers 
for measuring changes, and (4) is not limited to measuring change in 
oral communication, but may be applied to any classroom, industry, or 
organizational structure where behavior is the object of measurement. 
(JM) 



1 6005003 



Dr. Ronald L. Applbaum 

Department of Speech Communication 

California State College, long Beach 

6101 E. Seventh 

Long Beach, California 90801 



U.S. DEPARTMENT OP HEALTH. EDUCATION 
ft WELFARE 
OFFICE OF EDUCATIDN 
THIS DOCUMENT HAS BEEN REPRODUCED 
EXACTLY AS RECEIVED FROM THE PERSON OR 
ORGANIZATION ORIGINATING IT, POINTS OF 
VIEW OR OPINIONS STATED DO NOT NECES- 
SARILY REPRESENT OFFICIAL OFFICE OF EDU- 
CATION POSITION OR POLICY 



RATING SPEECH BEHAVIOR CHANGE OF STUDENTS: 
OBJECTIVIZING SUBJECTIVE JUDGMENTS 
by 

Dr. Ronald L. Applbaum 



0 

0 




When the behavioral scientist studies human behavior he is concerned 
with the internal and/or external aspects of what men do and do not do. 

3h this article I will restrict the meaning of behavior and behavior change 
to include only response or movement which is observable to the individual 
TTinlfing the judgment of change, e.g., a teacher views a student responding 
differently to the same stimulus after a unit of instruction. 

In the field of speech, teachers are concerned with both the behavior 
of students at one point in time, e.g., a student’s behavior or performance 
during an infoimational speech, and a change in behavior manifested by 
students over a period of time, i.e., a student’s behavior in speaking 
situations should change as a result of a particular course of instruction. 
While rating forms, check sheets, and numerous grading procedures have been 
utilized for assessing student behavior, these methods have not always 
proven highly reliable even in laboratory situations. The aforementioned 
behavioral measures were developed and utilized for measuring behavior at 
one point in time and their reliability may be influenced by extending 
the time between measurements. The purpose of this article is threefold; 

1. to explain a method for assessing, reliably, the behavioral change of 
speech students. The method was developed for measuring behavioral change 
and not measurement of behavior at one point in time. It was concerned with 
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creating consistency in change score measurements} 2. to present the results 
of research conducted utilizing this measurement procedure} 3« to illuminate 
possible implications of this methodology for measuring behavioral change. 

METHOD AND PROGEDUl^ 

Method 

Because a teacher’s perceptual field of the commionicative process is 
the sum or product of his experiences irlth communication encoimters, each 
teacher views students’ communicative behaviors differently. When rating 
student behavior teachers use both the measuring instrument, rating 

forms, and prior experiences in making their judgments. The evaluation of 
student behavior is in part a reflection of the teacher’s experiences. 
Differences in speech behavior as rated hy two teachers may reflect differ- 
ences in teachers’ perceptions, rather than behavioral differences among stu- 
dents. The individual teacher when evaluating a student’s communicative 
behavior may rate differently at two rating periods because of changes 
occuring within his own perceptual field, rather than any student behavioral 
change. The teachers’ perceptual field becomes the primary measuring instru- 
ment in rating behavior. The rating form used by the teacher only fixes 
subjective categories to which teacher bias is applied. The instruments 
commonly used to judge behavior make no attempt to control the influence 
of the measurers perceptual field during the measurement process. The 
method under discussion attempts to control the influence of a teacher’s 
perceptual field upon the evaluation process. 

When using this method the teacher becomes the measuring instrument 
because his experiences are used for evaluating student behavior. The 
teacher anchors his behavioral experiences on a forced-choice continuum. 

The continuum represents a distribution of particular behaviors, e.g., 
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coiranimicative behaviors, and categories or groupings of related behaviors 
are positioned on the continuum, e.g., poor communicative behaviors could 
represent an end category on a continuum of behaviors associated with 
communicative effectiveness. The teacher selects an individual from his 
experience who demonstrated the category of behaviors. The individual 
symbolizing the category of behaviors serves as a permanent anchor through- 
out the evaluation process. To extend oTir earlier example, if a teacher 
anchors the poor communicative behaviors, he selects an individual from 
his experience whose behaviors correspond to those in the poor communi- 
cative behavior category. This individual becomes the image for that 
category during the rating period. The procedure of anchoring image 
persons to categories is repeated for each category along the behavioral 
continuum, e.g., in this study five categories of behaviors associated 
with communicative effectiveness were anchored by the raters. 

To measure student behavior the teacher compares the individual 
anchors to each student’s communicative behavior. Subsequent evaluations 
by the teacher utilize the same anchors on the continuum. Thus students rated 
can move up or down along the continuum and a measurement of behavioral 
change can be made. The method is subjective because it uses the prior 
experiences of teachers as bases for comparing student behaviors, but it 
objectivizes the teacher’s rating system by providing a consistent scale 
for his observations over a period of time. 

Procedures 

The method was tested in both the elementary and secondary level. Eight 
classes, two eighth (28 Ss each) and two eleventh (21 Ss) were randomly 
selected for testing on the secondary level, two second (24 SB each) and 
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(26 Ss each) were randomly selected on the elementary level. Twenty 
raters were selected and divided into two groups. One group labeled ex- 
perienced judges contained ten raters having teaching experience and 
major/minor in speech. A second group labeled inexperienced judges con- 
tained ten raters without prior speech courses or teaching experience. 

Two experienced and two inexperienced judges were randomly selected for 
each class level. 

Raters were trained in the methods use at a special laboratory con- 
ducted at Pennsylvania State University. A five category scale of "com- 
municative behavior” was used to test the method. "Communicative behavior” 
was defined as any observable behavior manifested by students during an 
oral communicative situation in the classroom. A“ list of communicative 
behaviors characteristic of the subjects tested was collected from a 
survey conducted during a Title III, Area J, Oral Coram'uni cations Summer 
Institute under the guidance of Dr. Gerald M. Phillips. During the train- 
ing period, judges were instructed to anchor five categories, poor, average, 
excellent, above-average, and below average, drawn from the survey. Fig. 1 
illustrates the scale categories used as anchor points in the study. For 
example, Jerry represents the anchor or image person in the judge’s per- 
ceptual field who has demonstrated the behaviors in the category labeled 
"excellent communicative behaviors.” Jerry’s image is used by the judge 
to compare with students* actual behavior during the rating period. 

Fig. 1 
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Category 1 


Category 2 


Category 3 


Category 4 


Category 5 



Poor Below Average Above Excellent 

Average Average 
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Each set of four judges observed a randomly selected classroom for 
two rating periods. Six weeks elapsed between the initial and final rating 
periods. During the six week intei^al teachers were asked not to vary 
their usual classroom procedures. All teachers on the secondary level 
and two on the elementary level utilized for the test were trained in 
the use of oral communication techniques in the classroom and had begun 
implimenting the techniques prior to the judges first observations. In 
addition the teachers were not told the actual purpose of the research. 

The judges were not retrained after the first rating period or given an 
opportunity to review their initial ratings. The difference score between 
the judges* two ratings of a student’s communicative behavior constituted 
the behavioral change index. 

Hypotheses 

Two hypotheses were tested in this study; 

1. There are differences between experienced and inexperienced 
judges* measurement of communicative behavioral change. 

2. There are significant similarities among the four raters* 
measurements of students* communicative behavioral change. 

Hypothesis one was tested using a one factor analysis of variance-repeated 

measures to find if experienced judges rated behavioral change differently 

than inexperienced judges. Hypothesis two tested if the judges* behavioral 

change measurements were related and, thus, reliable indices of behavioral 

change. An intra-class correlation coefficient calculated from a one factor 

analysis of variance repeated measures tested the similarities of change 

scores. 

Results 

Table 1 illustrates the statistical results of hypotheses one and two. 
Because judge and student mortality rates exceeded acceptable limits, the 
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data in one eleventh grade class was not analyzed. The results of the 
test for hypothesis one in the remaining seven classes indicated no signi- 
ficant differences between judges’ measurements in any of the class levels, 
i.e., no statistically significant differences existed between experienced 
and inexperienced judges. Hypothesis one was not supported. The results 
of the test for hypothesis two indicate judges mean reliability ranging 
from .85 to a low .24. Judges’ reliability on the secondary level exceeded 
those on the elementary level in all but one classroom. The results support 
hypothesis two in all but two elementary classes. The judges’ measurements 
were reliable indices of ’’communicative behavior” change. 

TABLE 1 

Rater Reliability 



Elementary 


Secondary 


Class 

Level MS F r 


Class 

Level MS F r 


2 - A 1.97 .75 

B .49 

C .35 .71* 

D .49 


8 - A 1.66 .79 

B .35 

C .07 . 19 * 

D .36 


2 - A 1.06 .24 

B .80 

C .15 .18* 

D .85 


8 - A 1.64 .85 

B .24 

c .40 1 . 74 * 

D .23 


5 - A .94 .60 

B .27 

C .20 .54* 

D .37 


11 - A .35 .70 

B .11 

C .30 3.0* 

D .10 


5 - A .66 .44 

B .37 

C .20 . 52 * 

D .38 


11 - Not analyzed due to judge 
and subject mortality 



A = between subjects * nonsignificant at .05 level 

B = within subjects 
C = between judges 
D = residual 
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Discussion 

The results appear to confirm that the method can provide educators 
with a reliable means for assessing the attainment of behavioral change. 

By transforming educational objectives Into their behavioral counterparts 
the educator can use the method to measure students’ change in behavior, 
hence achievement of the objectives. The students’ behavioral change, 
or lack of It, could reflect the success or failure of a particular 
teaching method. 

The results indicate that the method may function more reliably on 
the secondary level. This may be attributed to three factors. First, 
raters may find it easier to develop anchors from experiences with secondary 
level students. Second, the communicative behaviors manifested on the 
elementary level may not be as easily categorized as those on the secondary 
level. Third, the two classes in which the reliabilities were particularly 
lew were both taught by teachers who restricted their students’ communicative 
behavior, thus, restricting the entire rating process. 

The method prepares outside observers for measuring change in the 
classroom. The only prerequisite to being a rater is that one have a 
minimum of experience with the behavior, otherwise, the creation of anchor 
images will be impossible. Disinterested observers, consultants, and 
students, can now be trained to make consistent judgments, e.g., students 
could observe other students and these ratings could be compared; teachers, 
disinterested observers, and consultants can evaluate classroom processes 
and make the proper response; teacher and peer ratings can be compared. 

The method is, of course, not limited to measuring change in oral 
communicative behavior. It is applicable to any classroom, industry, or 
organizational sit\iation where behavior is the object of measurement • If 
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the behaviors can be positioned on a continuum, they are amenable to 
measurement. The behavior measured can be either general or specific 
depending on the needs of the teacher. It is possible that several be- 
havioral objectives can be measured at one time, e.g., anxiety, attentive- 
ness, verbal structure. Teachers may discover that behaviors have differing 
rates of change and related behaviors need differing strategies to produce 
optimum behavioral change. 

The timing of measurement is flexible. Teachers can rate students 
at any desired time. Thus, teachers can assess both long range and short 
tenn effects of their teaching practices. The flexibility enables teachers 
to alter their strategies when rating results indicate that desired changes 
are not occuring. The flexibility may also illustrate that students’ be- 
havioral change occiirs at specific intervals during the use of a particular 
teaching strategy. 

There appear to be three limitations to the use of this methodology. 
First, if the behavioral categories are not sufficiently specific, the 
raters may find it difficult to develop anchor images. Second, raters 
totally unfamiliar with the behaviors to be measured would find it impos- 
sible to establish anchor images because they could not derive individuals 
from their experience to serve as anchor points. Third, the ?nethod assiames 
that behaviors in a category will remain constant throughout the rating 
periods. Certain observable behaviors cannot be rated because they are 
specific to the situation being rated, e.g., audience adaptation. The 
method measures change in behavior over a period of time and place. If 
the behaviors along the continuum are constantly changing their relative 
position, it would be incorrect to attempt an application of this method. 
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